Systems and methods for intelligently calibrating infrastructure devices using onboard sensors of an autonomous agent

ABSTRACT

A system for intelligently implementing an autonomous agent that includes an autonomous agent, a plurality of infrastructure devices, and a communication interface. A method for intelligently calibrating infrastructure (sensing) devices using onboard sensors of an autonomous agent includes identifying a state of calibration of an infrastructure device, collecting observation data from one or more data sources, identifying or selecting mutually optimal observation data, specifically localizing a subject autonomous agent based on granular mutually optimal observation data, identifying dissonance in observation data from a perspective of a subject infrastructure device, and recalibrating a subject infrastructure device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.17/011,037, filed 3 Sep. 2020, which is a continuation of U.S.application Ser. No. 16/792,780, filed 17 Feb. 2020, which claims thebenefit of U.S. Provisional Application Ser. No. 62/806,564, filed 15Feb. 2019, which is incorporated herein in its entirety by thisreference.

TECHNICAL FIELD

The inventions relate generally to the vehicle automation field, andmore specifically to new and useful systems and methods forintelligently arbitrating modes of operating an autonomous agent in thevehicle automation field.

BACKGROUND

State of the art vehicle automation presently enables some autonomousvehicles to operate in a substantially and/or fully autonomous state. Anability of such autonomous vehicles to operate effectively and safely inbusy or active environments often relies on an ability of the autonomousagent to observe its operating environment and make operating decisionsthat enables the autonomous agent to achieve a routing or traveling goalin a safe manner.

To observe an operating environment, autonomous agents may be configuredwith many sensors, sources of compute, sources of power and actuatorsthat operate together to enable the autonomous vehicle to perceive andcomprehend its environment and further, compute decisions and controlsthat optimizes how safely that the autonomous vehicle traverses theenvironment.

A safe operation of some autonomous agents often relies on accurate andcalibrated sensors that enable the autonomous agents to intelligentlyand safely evaluate its environment and driving conditions. However,over time, some characteristic of these sensors may change in terms ofposition and/or degradation. Accordingly, these sensors may becomeuncalibrated and therefore, diminish a quality of sensing data providedto the autonomous agent.

Thus, there is a need in the vehicle automation field for systems andmethods for intelligently calibrating sensors and, namely,infrastructure sensors that enhanced comprehension of an operatingenvironment by an autonomous agent. The embodiments of the presentapplication described herein provide technical solutions that address,at least, the needs described above.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 illustrates a schematic representation of a system forimplementing an autonomous agent in accordance with one or moreembodiments of the present application;

FIG. 2 illustrates an example schematic representation of an autonomousagent operating system in accordance with one or more embodiments of thepresent application;

FIG. 3 illustrates an example method in accordance with one or moreembodiments of the present application;

FIG. 4 illustrates an example schematic of a subsystem of the system 100for identifying uncalibrated infrastructure devices and generatingcalibration parameters in accordance with one or more embodiments of thepresent application;

FIG. 5 illustrates a variation of the method 200;

FIG. 6 illustrates an embodiment of the method 200;

FIG. 7 illustrates a variation of the method 200 performed with avariation of the system 100; and

FIG. 8 illustrates a variation of a set of infrastructure devices.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The following description of the preferred embodiments of the presentapplication are not intended to limit the inventions to these preferredembodiments, but rather to enable any person skilled in the art to makeand use these inventions.

1. Overview

As shown in FIGS. 1-2 , a system 100 for intelligently implementing anautonomous agent that includes an autonomous agent 110, a plurality ofinfrastructure devices 120, and a communication interface 130, asdescribed in U.S. Provisional Application Nos. 62/701,014 and62/718,642, which are both incorporated herein in their entireties bythis reference.

As shown in FIG. 5 , a method 200 for intelligently calibratinginfrastructure (sensing) devices (e.g., sensors, infrastructure devices,etc.) using onboard sensors of an autonomous agent includes identifyinga state of calibration of an infrastructure device S205, collectingobservation data from one or more data sources, S210, identifying orselecting mutually optimal observation data S220, specificallylocalizing a subject autonomous agent based on granular mutually optimalobservation data S230, identifying dissonance in observation data from aperspective of a subject infrastructure device S240, and recalibrating asubject infrastructure device S250.

2. Benefits

The system and/or method can confer several benefits over conventionalsystems and methods.

First, in some variations, the system and/or method confers the benefitof enabling a calibration error associated with an autonomous agentand/or its infrastructure to be identified. In a set of specificexamples, for instance, a comparison between observation data collectedat a set of sensors of an autonomous agent and observation datacollected at a set of sensors of an infrastructure device enables adetermination of a calibration error to be made.

Second, in some variations, the system and/or method confers the benefitof enabling a particular type of calibration error (e.g., intrinsiccalibration error as described below, extrinsic calibration error asdescribed below, etc.) to be identified. In a set of specific examples,for instance, different types of information or data associated witheach potential source of error (e.g., each sensor) are assessed. Healthdata and observational data, for instance, from each sensor of aninfrastructure device can be assessed in an event of a detectedcalibration error. In another set of specific examples, historicalinformation associated with a set of sensors is used to determine whichcalibration error is most likely to have occurred. In yet another set ofspecific examples, the type of calibration error is predicted (e.g.,using a set of algorithms, using a deep learning model, etc.).

Third, in some variations, the system and/or method confers the benefitof enabling a particular source (e.g., infrastructure device versusautonomous agent, particular sensor of infrastructure device, particularsensor of autonomous agent, etc.) of calibration error to be identified.

Fourth, in some variations, the system and/or method confers the benefitof enabling a particular time in which a calibration error occurred tobe identified.

Fifth, in some variations, the system and/or method confers the benefitof enabling a remote update of a calibration of at least one of aninfrastructure device and an autonomous agent. In a set of specificexamples, for instance, upon determining a calibration error at a remotecomputing system, a calibration update is automatically and wirelesslytransmitted to an identified source or sources of the calibration error.This can in turn function, for instance, to enable the operation of aself-sufficient set of infrastructure devices, which can augment anautonomous agent's perception of its surroundings.

Sixth, in some variations, the system and/or method confers the benefitof leveraging spatially aware infrastructure devices to maintain anoverall calibration of the system. As such, this can additionallyfunction to detect early calibration errors, determine a source of acalibration error (e.g., identify a vehicle calibration error versus aninfrastructure device calibration error, identify a particular sensor inneed of a calibration update, etc.), and/or perform any other suitablefunction. This can be advantageous over conventional systems and methodsincluding infrastructure devices, which are not spatially aware but usedmainly or solely for communication (e.g., strapped to a traffic lightand configured to communicate the color of the traffic light). Inspecific examples of this variation, the spatial awareness of theinfrastructure device enables calibration of the overall system to beperformed through detection and localization of the autonomous agent(s)within the data stream of the infrastructure device.

Seventh, in some variations, the system and/or method confers thebenefit of leveraging and/or adapting to existing infrastructure formounting of the infrastructure devices. In specific examples in whichthe infrastructure devices are mounted at various heights (e.g., tointerface with existing infrastructure of different types and/orheights, to conform to local ordinances, to interface with publiclyowned poles, etc.), a lens of the infrastructure camera can be chosen inaccordance with the height of the camera.

Additionally or alternatively, the system and/or method can confer anyother suitable benefits.

1. System for Implementing an Autonomous Agent

As shown in FIGS. 1-2 , a system 100 for intelligently implementing anautonomous agent that includes an autonomous agent 110, a plurality ofinfrastructure devices 120, and a communication interface 130, asdescribed in U.S. Provisional Application Nos. 62/701,014 and62/718,642, which are both incorporated herein in their entireties bythis reference. As shown in FIG. 4 , a subsystem of the system 100includes a plurality of autonomous health monitors and an autonomousstate machine.

The system 100 can additionally or alternatively include any or all ofthe systems, components, and/or embodiments described in any or all of:U.S. patent application Ser. No. 16/514,624, filed 17 Jul. 2019, U.S.patent application Ser. No. 16/505,372, filed 08, Jul. 2019, and U.S.patent application Ser. No. 16/540,836, filed 14 Aug. 2019, each ofwhich is incorporated herein in its entirety by this reference.

The autonomous agent 110 preferably includes an autonomous vehicle 110that is preferably a fully autonomous vehicle, but may additionally oralternatively be any semi-autonomous or fully autonomous vehicle; e.g.,a boat, an unmanned aerial vehicle, a driverless car, etc. Additionally,or alternatively, the autonomous agent 110 may be a vehicle thatswitches between a semi-autonomous state and a fully autonomous state(or a fully-manned state) and thus, the autonomous agent 110 may haveattributes of both a semi-autonomous vehicle and a fully autonomousvehicle depending on the state of the autonomous agent 110. While someportions of the embodiments of the present application are describedherein as being implemented via an autonomous agent 110 (e.g., anautonomous vehicle (e.g., a driverless car), a semi-autonomous, anunmanned aerial vehicle (e.g., a drone), or the like) it shall be notedthat any suitable computing device (e.g., any edge device includingmobile computing devices, etc.) may be implemented to process sensordata of an autonomous agent 110. While it is generally described thatthe autonomous agent 110 may be an autonomous vehicle, it shall be notedthat the autonomous agent no may be any type of kind of autonomousmachine, autonomous device, autonomous robot, and/or the like.

In a preferred embodiment, the autonomous agent no includes an onboardcomputing system 115 (e.g., a computer integrated with the autonomousagent) or any suitable vehicle system but can additionally oralternatively be decoupled from the autonomous agent 110 (e.g., a usermobile device operating independent of the autonomous agent).

Additionally, or alternatively, the onboard computing system 115 caninclude a processing system (e.g., graphical processing unit or GPU,central processing unit or CPU, or any suitable processing circuitry) aswell as memory. The memory can be short term (e.g., volatile,non-volatile, random access memory or RAM, etc.) and/or long term (e.g.,flash memory, hard disk, etc.) memory. As discussed below, theautonomous agent 110 may additionally include a communication interface130 that includes a wireless communication system (e.g., Wi-Fi,Bluetooth, cellular 3G, cellular 4G, cellular 5G, multiple-inputmultiple-output or MIMO, one or more radios, or any other suitablewireless communication system or protocol), a wired communication system(e.g., modulated powerline data transfer, Ethernet, or any othersuitable wired data communication system or protocol), sensors, and/or adata transfer bus (e.g., CAN, FlexRay).

Additionally, or alternatively, the autonomous agent no may be inoperable communication with a remote or disparate computing system thatmay include a user device (e.g., a mobile phone, a laptop, etc.), aremote server, a cloud server, or any other suitable local and/ordistributed computing system remote from the vehicle. The remotecomputing system is preferably connected to one or more systems of theautonomous agent through one or more data connections (e.g., channels),but can alternatively communicate with the vehicle system in anysuitable manner.

In some variations, information from the infrastructure devices and theautonomous agents is received at a cloud computing system, whereprocessing of the information (e.g., to determine mutually optimalobservation data) is performed. Additionally or alternatively, any orall of the processing can be performed at an onboard computing system115, at processing onboard one or more infrastructure devices, anycombination of these locations, and/or at any other suitablelocation(s).

The onboard computing system 115 preferably functions to control theautonomous agent 110 and process sensed data from a sensor suite (e.g.,a computer vision system, LiDAR, flash LiDAR, wheel speed sensors,radar, GPS, etc.) of the autonomous agent 110 and/or other(infrastructure device 120) sensors to determine states of theautonomous agent no and/or states of agents in an operating environmentof the autonomous agent no. Based upon the states of the autonomousagent and/or agents in the operating environment and programmedinstructions, the onboard computing system 115 preferably modifies orcontrols behavior of autonomous agent no. Additionally, oralternatively, the onboard computing system 115 preferably includes amulti-policy decision-making module 114 that functions to generatebehavioral policies and select a behavioral policy that the onboardcomputing system 115 may function to execute to control a behavior ofthe autonomous agent no.

The onboard computing system 115 is preferably a general-purposecomputer adapted for I/O communication with vehicle control systems andsensor systems but may additionally or alternatively be any suitablecomputing device.

Additionally, or alternatively, the onboard computing system 115 ispreferably connected to the Internet via a wireless connection (e.g.,via a cellular link or connection). Additionally, or alternatively, theonboard computing system 115 may be coupled to any number of wireless orwired communication systems.

The infrastructure devices 120 preferably function to observe one ormore aspects and/or features of an environment. In such preferredembodiments, the infrastructure devices additionally function to collectdata associated with the observations and transmit the collected dataand/or processed derivatives of the collected data to the autonomousagent no and/or a remote computing system. Additionally, oralternatively, each of the infrastructure devices 120 may function togenerate health status data based on continuous and/or periodicintrospective evaluations of one or more operations, processes, operablefeatures, and the like of the respective infrastructure device 120. Theinfrastructure devices 120 may function to transmit the health statusdata separately from and/or along with sensed data (observation data),as described in U.S. Provisional Application No. 62/702,715, which isincorporated herein in its entirety by this reference. In someembodiments, the infrastructure devices 120 may be referred to herein asroadside units. The roadside units preferably include devices in animmediate and/or close proximity to an operating position of anautonomous agent no, such as an autonomous car, and may function tocollect data regarding circumstances surrounding the autonomous agent noand in areas proximate to a zone of operation of the autonomous agent110. In some embodiments, the roadside units may include one or more ofoffboard sensing devices including flash LiDAR, LiDAR, thermal imagingdevices (thermal cameras), still or video capturing devices (e.g., imagecameras and/or video cameras, etc.), global positioning systems, radarsystems, microwave systems, inertial measuring units (IMUs), and/or thelike.

The infrastructure devices 120 may additionally or alternatively includecomputing capabilities via processing circuitry and a communicationinterface that enables the infrastructure devices 120 to communicatewith an autonomous agent 110.

The infrastructure devices are preferably arranged throughout a set ofone or more predetermined routes associated with one or more autonomousagents. Additionally or alternatively, the infrastructure devices can bearranged to cover any suitable route, such as arranged in a grid orgrid-like fashion, and/or arranged in any other suitable arrangement.

In some variations, for instance, the infrastructure devices arerestricted (e.g., based on local ordinances) and/or desired to be placedon publicly-owned infrastructure, such as publicly-owned utility poles.As such, the spacing between nearest neighbor infrastructure devices mayvary among the infrastructure devices, and subsequently the region whichthe infrastructure devices must monitor can significantly vary in size(and in shape, etc.). Additionally or alternatively, the heights atwhich the infrastructure devices are mounted may vary among the poles,which can function to avoid obstacles on the poles, work aroundobstacles in the line of sight, and/or perform any other suitablefunction(s).

The infrastructure devices are preferably mounted to existinginfrastructure, such as any or all of: poles (e.g., utility poles),buildings, traffic lights and/or posts, stop signs or other road signs,sidewalks or roads, and/or any other suitable infrastructure.Additionally or alternatively, the infrastructure devices can be securedto custom components, such as custom infrastructure. Any or all of theinfrastructure can be publicly owned, privately owned, or anycombination. The infrastructure devices can be arranged such that theyare each at a uniform height, such as a predetermined constant height.Additionally or alternatively, any or all of the infrastructure devicescan be at various heights, such as to avoid line of sight obstacles.

The particular sensor(s) and/or sensor components can optionally beselected based on the spacing among infrastructure devices and/or theheight or other parameters associated with the ultimate location of theinfrastructure device (e.g., mounted position). In some variationswherein an infrastructure device includes a camera, for instance, thelens and/or focal length of the camera is chosen based on the relativeproximity of other infrastructure devices relative to this particularinfrastructure device (e.g., as shown in FIG. 8 ). In a set of specificexamples (e.g., when the infrastructure on which the infrastructuredevices are mounted is restricted), the lens of the camera for eachinfrastructure device is chosen based on the size of the region that thecamera must detect, which is based on the proximity of the nearestinfrastructure devices. For infrastructure devices which must view alarge area, a relatively short focal length lens can be chosen whereasfor infrastructure devices which can view a small area, a larger focallength can be chosen. Additionally or alternatively, the lens can beadjusted based on a relative importance of the region being monitored(e.g., contains an important intersection, is highly trafficked bypedestrians, etc.). Further additionally or alternatively, any otherfeatures of a camera (e.g., resolution, sensor type such as a fullframe, etc.) or other infrastructure device sensor (e.g., radar, LiDAR,etc.) can be chosen.

A technical benefit achieved by the implementation of the infrastructuredevices 120 includes an ability to observe circumstances (e.g., aroundcorners, down perpendicular streets, etc.) beyond the observable scopeof the autonomous agent 110. The autonomous agent 110 may function toaugment data derived by its own onboard sensor suite with the additionalobservations by the infrastructure devices 120 (e.g., the roadside) 120to improve behavioral policy selection by the autonomous agent 110.

Additionally, or alternatively, in various embodiments theinfrastructure devices 120 are able to detect and track any type or kindof agents in an operating environment, such as with a video camera orradar. In such embodiments, an example video camera may function toprovide detection of agents and semantic classification of the agenttype and possible intent of an agent, such as a pedestrian that is aboutto cross a road, or a car that is about to make a left turn, a driverwhich is about to open a car door and exit their vehicle and/or thelike.

In a first set of variations of the set of infrastructure devices, eachinfrastructure device includes a set of one or more cameras, wherein theset of cameras can individually and/or collectively detect an autonomousagent (e.g., coarsely localize, precisely localize, etc.) and optionallyany number of features within a route (e.g., predetermined route,potential route, etc.) of the autonomous agent, such as any or all of:road infrastructure (e.g., intersections, stop signs, lane markings,streets, etc.); pedestrians; other vehicles (e.g., vehicles proximal theautonomous agent) or modes of transportation (e.g., bicycles, scooters,etc.); and/or any other suitable information.

In a second set of variations of the set of infrastructure devices, eachinfrastructure device includes a set of one or more LiDAR sensors, whichcan function to determine a precise, 3-dimensional localization of anautonomous agent. In specific examples, the LiDAR sensors of theinfrastructure device can function to know an autonomous agent'sposition with greater position than the autonomous agent itself.

Additionally, or alternatively, other infrastructure devices 120 mayinclude traffic management devices or the like operating in theenvironment that may function to communicate with one or more of theroadside units and/or communicate directly with the autonomous agentregarding data collected and/or sensed by the infrastructure device,regarding an operating state of the infrastructure device (e.g., red orgreen traffic light), and the like. For example, in the case that theautonomous agent 110 is an autonomous vehicle, a traffic light may be aninfrastructure device in an environment surrounding the autonomousvehicle that may function to communicate directly to the autonomousvehicle or to a roadside unit that may be in operable communication withthe autonomous vehicle. In this example, the traffic light may functionto share and/or communicate operating state information, such as a lightcolor that the traffic light is projecting, or other information, suchas a timing of the light changes by the traffic light, and/or the like.

The communication interface 130 preferably enables the autonomous agent110 to communicate and/or exchange data with systems or devices externalto the autonomous agent 110. Preferably, the communication interface 130enables one or more infrastructure devices 120 to communicate directlywith the autonomous agent 110. The communication interface 130preferably includes one or more of a cellular system (or any suitablelong-range communication system), direct short-wave radio, or any othersuitable short-range communication system. Additionally, oralternatively, the communication interface 130 may function to enableinter-vehicle communication between the autonomous agent no and othervehicles, including fully autonomous vehicles, semi-autonomous vehicles,and manual vehicles.

In some embodiments, in addition to a powertrain (or othermovement-enabling mechanism), autonomous agent no may include a sensorsuite (e.g., computer vision system, LiDAR, radar, wheel speed sensors,GPS, cameras, etc.) or onboard sensors that are in operablecommunication with the onboard computing system 115.

The sensor suite preferably includes sensors used to perform autonomousagent operations (such as autonomous driving) and data capture regardingthe circumstances surrounding the autonomous agent no as well as datacapture relating to operations of the autonomous agent no but mayadditionally or alternatively include sensors dedicated to detectingmaintenance needs of the autonomous agent 110. For example, the sensorsuite may include engine diagnostic sensors or an exterior pressuresensor strip. As another example, the sensor suite may include sensorsdedicated to identifying maintenance needs related to cleanliness ofautonomous agent interiors; for example, internal cameras, ammoniasensors, methane sensors, alcohol vapor sensors, etc.

In preferred variations, the sensor suite of the autonomous agentincludes multiple types of sensors, such as, but not limited to, any orall of: cameras, LiDAR sensors, radar sensors, and GPS sensors.

Any or all of the sensors of the sensor suite can be placed onboard theautonomous agent (e.g., on an external surface, on an internal surface,etc.); offboard the autonomous agent (e.g., arranged remotely, as partof a secondary device such as a mobile device, etc.); and/or at anycombination of the two.

In some variations, at least a subset of sensors (e.g., cameras, LiDARsensors, radar sensors, etc.) are mounted to an exterior surface of anautonomous agent vehicle (e.g., doors, bumper, roof, windows,windshield, etc.). Additionally or alternatively, at least a subset ofsensors, which can include the same type of sensors and/or differentsensors as those described above for being mounted to an externalsurface of the autonomous agent vehicle (e.g., mounted to the vehicle'sframe, attached to a windshield/roof/other surface, etc.), are mountedto an interior surface of the autonomous agent (e.g., inside of doors,seats, ceiling, floor, windshield, etc.). In specific examples, forinstance, multiple sensors are arranged at numerous locations on theexterior and within the interior of the vehicle such that the sensorsare able to capture a desired region. The desired region can be definedby, but not limited, any or all of: a desired field-of-view (e.g., adesired predetermined portion of a route, a 360-degree field-of-view, avertical field-of-view such as a height above ground that the sensorscan capture, etc.); a desired capture (e.g., in terms of resolution) ofone or features along a route (e.g., intersections, particularly riskylocations, pedestrians, road markings, unexpected features such as anaccident or construction site, etc.); a desired capture of one or moreinfrastructure devices (e.g., a range of heights that the sensors mustcapture); a predetermined geometric region; and/or any other suitableregions. It should be noted that in addition or alternative to a visualfield-of-view, “field-of-view” can equivalently refer to the rangecaptured by one or more radar sensors, the range captured by one or moreLiDAR sensors, and/or any other suitable ranges captured by any suitablesensors.

In some variations, especially for instances in which the autonomousagent vehicle is used to transport one or more passengers (e.g., anautonomous shuttle, an autonomous personal use vehicle, an autonomousride share vehicle, etc.), the set of potential calibration errors caninclude disturbances occurring in the routine use of a vehicle. Thesecan include any or all of: repeated closing and opening of doors (e.g.,driver door, passenger door, etc.); repeated closing and opening of atrunk; cleaning of the vehicle (e.g., cleaning a windshield, carwash ofthe external surfaces, cleaning/vacuuming of the interior, etc.);bumping of interior sensors; minor collisions; and/or any otherperturbations. Additionally or alternatively, the set of potentialcalibration errors can include normal wear-and-tear and/or exposure tothe environment (e.g., sun, rain, wind, etc.).

In a set of specific examples, at least a set of cameras are arranged atan interior and an exterior of a vehicle, and optionally one or both ofa set of LiDAR sensors and a set of radar sensors are further arrangedat at least the exterior of the vehicle and optionally the interior,such that the sensors are able to capture a large range of thesurroundings of the vehicle (e.g., 360-degree field-of-view withvertical height able to capture all infrastructure devices along route).

In accordance with one or more embodiments, an autonomous operatingsystem may generally include a controller 116 that controls autonomousoperations and/or actions of the autonomous agent 110, as shown by wayof example in FIG. 2 . That is, suitable software and/or hardwarecomponents of controller 116 (e.g., processor and computer-readablestorage device) are utilized to generate control signals for controllingthe autonomous agent 110 according to a routing goal of the autonomousagent no and selected behavioral policies of the autonomous agent.

Additionally, or alternatively, the autonomous agent no includes asensor fusion system 117, a positioning system 118, and a guidancesystem 119. As can be appreciated, in various embodiments, theinstructions may be organized into any number of systems (e.g.,combined, further partitioned, etc.) as the disclosure is not limited tothe present examples.

In various embodiments, the sensor fusion system 117 synthesizes andprocesses sensor data and predicts the presence, location,classification, and/or path of objects and features of the environmentof the autonomous agent 110. In various embodiments, the sensor fusionsystem 117 may function to incorporate data from multiple sensors and/ordata source, including but not limited to cameras, LiDARs, radars,infrastructure devices, remote data feeds (Internet-based data feeds),and/or any number of other types of sensors.

The positioning system 118 preferably processes sensor data along withother data to determine a position (e.g., a local position relative to amap, an exact position relative to lane of a road, vehicle heading,velocity, etc.) of the autonomous agent 110 relative to the environment.The guidance system 119 processes sensor data along with other data todetermine a path for the vehicle 110 to follow.

In various embodiments, the controller 116 may function to implementmachine learning techniques to assist the functionality of thecontroller 116, such as feature detection/classification, obstructionmitigation, route traversal, mapping, sensor integration, ground-truthdetermination, and the like.

The plurality of autonomous health monitors preferably function toevaluate data streams from any data source associated with an operationand/or control of an autonomous agent no and output autonomous healthintelligence data. In one embodiment, autonomous health intelligencedata may relate to a judgement by each of the autonomous health monitorsregarding a state of health of each of the devices and/or operationsthat produce the data streams. In one embodiment, each of the pluralityof autonomous health monitors may be configured to evaluate a differentdata stream originating from different data sources. For instance, afirst data stream from a first device may be evaluated by a firstautonomous health monitor and a second data stream from a second processor operation may be evaluated by a second autonomous health monitor andthe like.

Each of the plurality of autonomous health monitors may be in operablecommunication with the autonomous state machine. The autonomous statemachine preferably functions to consume the autonomous healthintelligence data as well as state data from any suitable source (e.g.,from the plurality of autonomous control planning modules 410 or thelike). In a preferred embodiment, the autonomous state machine may be adevice that is separate and distinct from the onboard computer 115 andmay function to calculate an operating state or a level of operability(e.g., runlevel) of an autonomous agent based on inputs of health data.The operating state and/or the level of operability preferably includesa value that indicates an extent to which the capabilities and/orfunctionalities of autonomous agent are operable or not operable (e.g.,runlevel 7=100% operability, runlevel 6=90% operability, and/or thelike). While it may be generally described that a given level ofoperability may be associated with a percentage level of operability ofan autonomous agent, it shall be noted that a level of operability shallnot be restricted to such example, but may additionally or alternativelyinclude embodiments in which a health system (e.g., health monitors) ofthe autonomous agent evaluates capabilities and/or operability of theautonomous agent's systems, and identifies a level of operability basedon this evaluation.

It shall be noted that while the autonomous state machine may beimplemented independently of an onboard computer 115 of an autonomousagent 110, the autonomous state machine may be integrated with anysuitable device and/or system of an autonomous agent 110 including theonboard computer 115.

The plurality of autonomous control planning modules 410 preferablyfunction to generate a plurality of autonomous control instructions foran autonomous agent 110. Accordingly, each of the plurality ofautonomous control planning modules 410 may function to output differentautonomous control instructions or plans that correspond to differentlevel of operability of an autonomous agent.

Additionally, each of the plurality of autonomous control planningmodules 410 may be configured to continuously collect data streams froma plurality of data sources (e.g., onboard and offboard data sources)and run in parallel to continuously simulate one or more trajectoryoutcomes for an autonomous agent 110 and correspondingly, generateautonomous control instructions based on a simulation having a highestprobability of occurring, as described in U.S. Provisional ApplicationNo. 62/701,014, which is incorporated by this reference in its entirety.

2. Method for Intelligently Calibrating Infrastructure Devices UsingInfrastructure Device Data and Autonomous Agent Data

As shown in FIG. 5 , a method 200 for intelligently calibratinginfrastructure (sensing) devices (e.g., sensors, infrastructure devices,etc.) using onboard sensors of an autonomous agent includes identifyinga state of calibration of an infrastructure device S205, collectingobservation data from one or more data sources, S210, identifying orselecting mutually optimal observation data S220, specificallylocalizing a subject autonomous agent based on granular mutually optimalobservation data S230, identifying dissonance in observation data from aperspective of a subject infrastructure device S240, and recalibrating asubject infrastructure device S250.

At least a portion of the method 200 is preferably performedcontinuously throughout the traversal of a route by an autonomous agent,such as at any or all of: a predetermined frequency, a random set ofintervals, and/or at any other suitable times. Additionally oralternatively, any or all of the method 200 can be performed in responseto a trigger (e.g., a time elapsed since last calibration update, aninput from a user, a detected state of calibration or uncalibration inS205, an error in driving such as an accident, a manual takeover of thevehicle, etc.), once per route traversal, greater than once per routetraversal, and/or at any other suitable time(s).

The method 200 can additionally or alternatively include any or all ofthe methods, processes, and/or embodiments described in any or all of:U.S. patent application Ser. No. 16/514,624, filed 17 Jul. 2019, U.S.patent application Ser. No. 16/505,372, filed 8 Jul. 2019, and U.S.patent application Ser. No. 16/540,836, filed 14 Aug. 2019, each ofwhich is incorporated herein in its entirety by this reference.

In some variations for instance (e.g., as shown in FIG. 3 ), the method200 can additionally include and/or interface with a method includingany or all of: collecting autonomous agent control data; identifying atakeover control signal; arbitrating active control signals; andtransitioning control of the autonomous agent. Additionally oralternatively, the method 200 can include any other suitable processes.

2.05 State of Calibration

The method can optionally include S205, which includes identifying astate of calibration of an infrastructure device, may function toidentify a calibration status of an infrastructure device arranged in anoperating environment of an autonomous agent. The calibration statuspreferably includes calibration status data indicating whether a subjectinfrastructure device is accurately calibrated or inaccuratelycalibrated. Additionally, or alternatively, the calibration status datamay include data indicating that a calibration status of a subjectinfrastructure device is unknown, indeterminable, and/or the like.

In the circumstances in which calibration status data indicates that asubject infrastructure device may be uncalibrated, the calibrationstatus data may additionally or alternatively indicate a (un)calibrationstatus type. That is, in addition to an indication that the subjectinfrastructure device is uncalibrated, the calibration status dataobtained from the subject infrastructure device may include additionalmetadata relating to one or more a plurality of distinct types ofuncalibration and/or one or more of a plurality of distinct error codesrelating to a malfunction or uncalibration of the subject infrastructuredevice.

For instance, a first type of uncalibration may relate to one or moretypes of extrinsic calibration errors, shifts, drift, and/or the like ofan infrastructure device. In one example, an extrinsic uncalibration mayrelate to a physical characteristic of an infrastructure device that maybe uncalibrated. In such example, a physical position and/or orientationof one or more physical features of the infrastructure device may beuncalibrated; meaning that the physical feature has shifted from a firststate of calibration and/or position to a second state of uncalibrationand/or position. In some variations, for instance, the position of aninfrastructure device on a utility pole (e.g., telephone pole) can bemoved/bumped by a worker doing maintenance for other objects (e.g.,electrical lines) associated with the infrastructure.

For variations of the infrastructure device including LiDAR sensorsand/or any other sensors with a mechanical component that spins overtime, a calibration error in the form of drift can occur due to spinningover time.

Accordingly, an extrinsic uncalibration of an infrastructure device canbe an uncalibration type that may be perceivable by an inspection (e.g.,by viewing, etc.) and/or evaluation of the external characteristics of asubject infrastructure device. However, it shall be noted that anextrinsic uncalibration may relate to any type of uncalibration of orrelating to a hardware component of an infrastructure device.

In another instance, a second type of uncalibration may relate to one ormore types of intrinsic calibration errors, shifts, and/or the like ofan infrastructure device. In one example, an intrinsic uncalibration mayrelate to an unexpected or unintended change(s) in the calibration ofone or more internal physical and/or software features of aninfrastructure device. In one example, if an infrastructure (sensing)device comprises a camera, an intrinsic uncalibration may relate to anunexpected shift or change in a focal length of the camera. In anotherexample, if an infrastructure device comprises a microwave sensor, anintrinsic uncalibration may relate to one or more software glitches thatmay affect the microwaves ability to properly compute or detect movementand/or compute a precise location of movement, etc.

Uncalibration of an infrastructure device may be caused by a varietyand/or combination of events or elements including, but not limited to,tampering, physical degradation caused by environmental factors (e.g.,nature, climate, etc.), defect(s) in characteristics of aninfrastructure device, drift over time, accidents, and/or the like. Itshall be recognized that the above-noted are merely examples and thetypes and/or causes of uncalibration of an infrastructure device causesof uncalibration and/or calibration errors in infrastructure devicesshall not be limited thereto.

In some implementations, a calibration status may be obtained oridentified based on health status data that is self-generated by aninfrastructure device in a normal course of operation of theinfrastructure device. For instance, an infrastructure device mayfunction to continuously and/or periodically provide (generate) healthstatus data that includes calibration status data. Additionally, oralternatively, health status data may be actively requested, or thegeneration of health status data may be triggered by one or more partiesand/or other devices (e.g., an autonomous agent, another infrastructuredevice, a remote-control terminal, etc.) that is in operablecommunication with the subject infrastructure device.

In a preferred implementation, a calibration status may be obtainedand/or identified according to one or more steps (e.g., S210-S240) ofthe methods and/or processes described herein.

In additional or alternative implementations, a calibration status isdetermined prior to the remaining processes of the method, wherein theresult of the calibration status is either used to trigger any or all ofthe remaining processes (e.g., in an event that the calibration statusindicates proper calibration) or prevent any or all of the remainingprocesses (e.g., in an event that the calibration status indicatesimproper calibration).

2.1 Obtaining/Receiving Infrastructure and/or Autonomous Agent Data

The method includes S210, which includes collecting observation datafrom one or more data sources, functions to collect streams ofobservation data from one or more data sources including one orinfrastructure devices and one or more autonomous agents that may enablea detection of an uncalibrated infrastructure device and a generation ofcalibration parameters. Generally, the collected data may includeobservation data, infrastructure device data (also referred to herein as“infrastructure data”), and/or autonomous agent data (sometimes referredto herein as “agent data”). Observation data may generally relate todata observed by and/or sensed by either or both of an infrastructuredevice and an autonomous agent. Infrastructure data may generally relateto data sourced from an infrastructure device that may includeobservation data. Autonomous agent data may generally relate to datasourced from an autonomous agent that may include observation data.

S210 is preferably performed continuously (e.g., at a predeterminedfrequency) throughout the method 200, but can additionally oralternatively be performed in response to S205, in place of S205, inabsence of S205, in response to a trigger, once during the method 200,and/or at any other suitable times. Further additionally oralternatively, the method 200 can be performed in absence of S210.

In a preferred embodiment, the one or more data sources may includedevices and/or systems of an autonomous agent, sensors mounted (e.g.,onboard sensors) on the autonomous agent, processes performed by orinvolving the autonomous agent, and infrastructure devices in aproximity of the autonomous agent. In one or more embodiments, datasourced from an autonomous agent (i.e., autonomous agent data)preferably includes precise location (e.g., GPS data over time and/orthe like) and agent model data that may include data that identifies theautonomous agent and that may also, identify physical features and/orattributes of the autonomous agent (e.g., an appearance of theautonomous agent). In a preferred embodiment, agent model data includesan expected model of the autonomous agent. The data collected in S210may additionally or alternatively include one or more health statusdata, which may include any or all of: operational status data,diagnostic information, resource utilization information, observationaldata (preferably collected from an environment or circumstancessurrounding a given infrastructure device or autonomous agent), temporalinformation (e.g., time since last calibration update), and/or any othersuitable information. The health status data may preferably includecurrent or prior calibration data of one or more sensors or of aninfrastructure device together with positional data, such as a two orthree-dimensional coordinate position (e.g., [x,y,z] position) thatidentifies a geographical position of an infrastructure device orautonomous agent.

Additionally, or alternatively, the infrastructure data and theautonomous agent data may be collected and provided as input into ashared compute resource, such as a calibration module and/or calibrationcircuit that determines a state of calibration and that may alsogenerate calibration parameters that enable a correction or remediationof a state of uncalibration of an infrastructure device or the like, asshown by way of example in FIG. 4 . The calibration module and/orcalibration circuit may be deployed in any suitable many, including viaa remote distributed computing system (e.g., the cloud), on a subjectautonomous agent, and/or at an infrastructure device.

It shall be noted that while the one or more data sources preferablyinclude devices and/or systems of the autonomous agent, onboard sensors(e.g., a sensor suite), and infrastructure devices (e.g., receivinghealth statuses and observation data from offboard devices, such asroadside units), the one or more data sources may additionally oralternatively include one or more remote data feeds (e.g., weather feed,traffic feed, etc.), a remote autonomous agent platform (e.g., remoteservers, cloud servers, etc. for remotely managing and/or operating anautonomous agent), other autonomous agents, and any other suitable datasources involved with or relating to an operation of an autonomous agentand/or one or more infrastructure devices.

In one preferred embodiment, S210 may function to collect streams ofdata from one or more onboard devices and/or onboard processesassociated with an autonomous agent. For example, one or more onboarddevices may include one or more onboard sensor devices that may functionto capture and/or sense data regarding an environment or circumstancessurrounding an autonomous agent. Additionally, or alternatively, the oneor more onboard devices may function to capture and/or sense datarelating to an operation of one or more low-level devices (e.g.,actuators, actuated devices, etc.), operating systems (e.g., onboardcomputing system, controller, sensor fusion system, positioning system,guidance system, communication interface, and the like), etc.

In one preferred embodiment, S210 may function to collect streams ofdata from infrastructure devices. In such preferred embodiment, S210functions to collect the streams of data during an operation of theautonomous agent but may also function to collect the streams of dataduring periods in which the autonomous agent is not in an active state(e.g., parked or the like). The infrastructure devices preferablyinclude one or more sensor devices that are intelligently arrangedand/or positioned within an (operating or driving) environment of anautonomous agent. For instance, the one or more sensor devices may bearranged to observe and/or collect data that may be assistive fordetermining and/or generating driving/operating (control) instructionsfor an autonomous agent and also, for decisioning by an autonomous agentwhen presented with multiple driving and/or operating instructions whichinstructions to execute and which instructions to disregard. Thus, theone or more infrastructure sensors may function to collect data in adriving environment, which may include road data, sidewalk data,short-range communication data (e.g., radio communication links, etc.),positions of static and/or dynamic object data (e.g., agent data),traffic data, and the like along a given route plan or a possible routeplan of a given autonomous agent.

In some embodiments, the infrastructure devices may include one or moresensor devices that are fixedly attached or positioned within an(driving) environment, such that a fixed (or substantially) coordinatelocation of the one or more sensor devices may be known when properlycalibrated. Accordingly, such fixedly arranged infrastructure devicesmay have a fixed field-of-detection. For instance, a camera fixed in adriving environment may have a fixed field-of-view. In some embodiments,the infrastructure devices may include one or more sensors devices thatare movably positioned within an environment, such that a coordinatelocation of the one or more sensor devices or physical features of theone or more sensor devices varies. In such embodiments, theinfrastructure devices may have a variable field-of-detection and may becapable of sensing data along multiple trajectories within anenvironment.

In a first implementation, S210 may function to automatically collectstreams of data from one or more infrastructure devices that are incommunication proximity (e.g., within a predetermined distance thresholdor a field-of-detection, etc.) of the autonomous agent. In someembodiments, the infrastructure devices may be configured to communicatewith an autonomous agent using short-ranged communication schemes orsystems. In such embodiments, once the autonomous agent has entered acommunication range or proximity of a given infrastructure device, theautonomous agent may function to automatically detect signals from theinfrastructure device and automatically collect data originating fromthe infrastructure device.

In a second implementation, S210 may function to automatically collectstreams of data from one or more infrastructure devices that are apredetermined distance from an operating autonomous agent. That is, insome embodiments, an operating environment of an autonomous agent mayinclude a plurality of infrastructure devices, however, the autonomousagent may be configured to automatically collect data from only a subsetof the plurality of infrastructure device within the predetermineddistance of the autonomous agent and possibly, ignore data incoming fromother infrastructure devices outside of the predetermined distance ofthe autonomous agent. In this way, the autonomous agent may function tocollect data having a more immediate or higher relative importance forpending and/or immediate operating decisions.

In a variant of the second implementation, S210 may function toautomatically collect streams of data from one or more infrastructuredevices that are a predetermined distance of the autonomous agent andweigh or consider data collected from the one or more infrastructuredevices within an active trajectory or travel path of the autonomousagent differently than data from infrastructure devices that are not orno longer within a trajectory or travel path of the autonomous agent.That is, in some embodiments, S210 may function to aggregate and weighdata from infrastructure devices that are substantially coincident witha position of the autonomous agent and ahead of a travel path of theautonomous agent with additional weight than a weight afforded to datafrom infrastructure devices behind or that has been passed by theautonomous agent along its travel path.

In some embodiments, the data collected from the one or moreinfrastructure devices may include compressed and/or semantically densedata regarding one or more features of an environment. In someembodiments, the field-of-detection of given infrastructure devicecomprises a geometrically defined region and within the geometricallydefined region, the infrastructure device may be configured to sense orcollect a semantic abstraction (e.g., a general shape or size,positions, velocity (moving or not moving) of the features, objects,and/or agents within the geometrically defined region.

Additionally, or alternatively, in some embodiments, the one or moreinfrastructure devices may be configured to sense or detect data withinthe geometrically defined region and derive and/or compute a state dataabout the circumstances within the geometrically defined shape. Forinstance, if the geometrically-defined shape or sensing region is asquare that includes a sidewalk or similar pedestrian path, a firstinfrastructure sensor device may function to identify whether there aremotionless agents (objects or persons that are not moving) and/or movingagents (objects or persons that are moving) on the sidewalk and provideas state data to the autonomous agent an indication confirmingmotionless agents or moving agents positioned on the sidewalk. In one ormore instances in the present application, motionless agents maysometimes be referred to herein as static agents thereby identifying anagent (object or person) that is not moving within a sensed region orscene. Also, in one or more instances in the present application, movingagents may sometimes be referred to herein as dynamic agents therebyidentifying an agent that is moving within a sensed region or scene.

In some cases, if there are no agents positioned within thegeometrically-defined sensing region, the state data may be anindication of no agents (e.g., “Clear”). Thus, in such embodiments,rather than sending a full representation of a scene within thegeometrically-defined shape or sensing region, the infrastructure devicemay provide semantically dense state data to an autonomous agent. As afew examples of state data, the infrastructure sensing devices mayindicate Agent or No Agent, Static Agent or Dynamic Agent, Clear or NotClear, Busy (Active) or Not Busy (Not Active), and/or any suitablesimplified and/or derivative information about circumstances within asensing region of an infrastructure device that may be provided to anautonomous agent.

2.2 Coarse Observation Selection

The method includes S220, which includes identifying or selectingmutually optimal observation data, functions to make a coarseidentification of observation data that preferably includes coincidencebetween infrastructure data of a subject infrastructure device andautonomous agent data of a subject autonomous agent. In a preferredembodiment, S220 may function to jointly evaluate infrastructure dataand autonomous agent data to coarsely identify segments ofinfrastructure data and segments of autonomous agent data in which thesubject infrastructure device and the subject autonomous agent may havemutually observed each other (via sensors and/or the like) during anormal course of operation. That is, S220 may function to determine oneor more instances in which it was probable or likely a subjectautonomous agent may have observed a subject infrastructure device andat a same time or period, the subject infrastructure device may haveobserved the subject autonomous agent. Accordingly, S220 enables acoarse localization of a subject autonomous agent as well as a subjectinfrastructure device within the mutually optimal observation data.

S220 is preferably performed continuously (e.g., at a predeterminedfrequency) throughout the method 200, but can additionally oralternatively be performed in response to S210, in place of any or allof the above processes, in absence of any or all of the above processes,in response to a trigger, once during the method 200, and/or at anyother suitable times. Further additionally or alternatively, the method200 can be performed in absence of S220.

In a first implementation, S220 preferably functions to identify coarsesegments (of mutually optimal observation data) of coincidence betweeninfrastructure data and autonomous agent data based on synchronizingtimestamp data obtained from both of the infrastructure data and theautonomous agent data. Additionally, or alternatively, S220 may functionto define the coarse segments of coincidence based on trajectory dataand/or position data (preferably timestamped) of the autonomous agentthat is preferably extracted from the autonomous agent data. Thetrajectory data, in some embodiments, may include a predeterminedtrajectory or route plan of a subject autonomous agent, which may havebeen identified from autonomous agent data. The trajectory data mayadditionally include expected position data, which may also includeexpected time data for each identified expected position of the subjectautonomous agent along the trajectory or route plan and expected timesat which the subject autonomous agent should arrive at each expectedposition along the trajectory. Additionally, or alternatively, thetrajectory data may include an actual or historical travel path taken bya subject autonomous agent, which includes position data along with timedata for each position of the autonomous agent along the travel path.

In a second implementation, S220 may function to identify coarsesegments of coincidence between infrastructure data and autonomous agentdata based on correlating position data of a subject autonomous agent toa field-of-detection of a subject infrastructure device. Preferably, inthis second implementation, S220 functions to identify one or moresegments of the infrastructure data in which the subject autonomousagent should have been found or positioned within a field-of-detectionof the infrastructure device based on the position data and timestampdata of the subject autonomous agent. In this way, S230 may function tocoarsely localize the autonomous agent at least within thefield-of-detection of the subject infrastructure device.

2.3 Observation Data Reduction & Granular Localization

The method can optionally include S230, which includes specificallylocalizing a subject autonomous agent based on granular mutually optimalobservation data, may function to identify one or more granularinstances of coincidence between infrastructure data and autonomousagent data in which a subject autonomous agent and a subjectinfrastructure device were most likely mutually observable and further,may function to precisely localize the autonomous agent within thefield-of-sensing of the infrastructure device based on the mutuallyoptimal observation data. That is, in one preferred embodiment, S230 mayfunction to specifically localize or identify one or more instances of asubject autonomous agent within the observation data obtained from asubject infrastructure device at the one or more instances in which itwas determined with high probability (e.g., beyond a localizationthreshold) that the subject autonomous agent and the subjectinfrastructure device may have been observing each other.

S230 is preferably performed continuously (e.g., at a predeterminedfrequency) throughout the method 200, but can additionally oralternatively be performed in response to S220, in place of any or allof the above processes, in absence of any or all of the above processes,in response to a trigger, once during the method 200, and/or at anyother suitable times. Further additionally or alternatively, the method200 can be performed in absence of S230.

Accordingly, granular mutually observation data preferably relates toobservation data in which a subject autonomous agent has beenspecifically identified or precisely localized from a sensing(observing) perspective of a subject infrastructure device. In onepreferred embodiment, S230 may function to achieve granular observationdata by reducing coarse observation data (as generated in S220) toeliminate noise or extraneous data to enhance a search capability and/orlocalization of the subject autonomous agent. That is, S230 may functionto extract from coarse mutually observation data a subset of observationdata comprising granular mutually observation data by paring awayextraneous data. For instance, coarse mutually observation data (CMOD)reduced by extraneous data should result in granular mutual observationdata (GMOD) (i.e., CMOD−Extraneous Data=GMOD).

Accordingly, in some embodiments, S230 may function to pare away and/orreduce from the coarse observation data extraneous data that may notpositively enable a localization and/or identification of a subjectautonomous agent within infrastructure data obtained from a subjectinfrastructure device. For instance, based on autonomous agent data thatmay include a model of the autonomous agent, such as a schematicrepresentation, an image, illustration, and/or any suitablerepresentation of the autonomous agent, S230 may function to pare awayor reduce any other autonomous agents, other vehicles, and/or any otherobject from the observation data (typically from the infrastructuredata) that does not include the given model of the autonomous agent. Inanother example, if observation data (sourced from the autonomous agent)indicates that the subject autonomous agent was dynamic and/or a movingobject during a period or instance of mutual observation, S230 mayfunction to pare or reduce from the observation data (sourced from theinfrastructure device) the data relating to static objects. In this way,once the coarse observation data is pared down to granular observationdata (i.e., data without extraneous data), the method 200 mayefficiently operate on and/or process the granular observation data toidentify and/or localize the subject autonomous data and determinewhether calibration of a subject infrastructure device is required.

Additionally, or optionally, the method 200 may function to pare down orreduce observation data including raw infrastructure data and rawautonomous agent data and/or the like ahead of performing S220. In thisway, S220 and S230 may not have to perform analysis of super extraneousobservation data either obtained from the infrastructure sensing devicesand/or the autonomous agent. For instance, if the method 200 involvescalibrating an autonomous agent which was not operating during aspecific window of time or period, the method 200 may function toeliminate or pare away infrastructure sensing data captured during thatperiod of time prior to performing an analysis of the observation data.Accordingly, super extraneous data preferably relates to observationdata in which the probability that a subset of the observation data isuseful or can be used to identify an autonomous agent or identify aninfrastructure device within the observation data for purposes ofcalibrating either the autonomous agent or the infrastructure device isbelow or does not satisfy a calibration data threshold. Accordingly, anysuitable data point from observation data may be used to determine aprobability or likelihood that a subset of observation data is superextraneous data and therefore, should not be considered and/oreliminated from the observation data prior to performing one or moreanalysis or processes of S220 and S230.

In a first implementation, once coarse mutual observation data is pareddown to granular mutual observation data, S230 may function tospecifically localize the subject autonomous agent within theinfrastructure data of a subject infrastructure device based onevaluating a given agent model of the subject of the autonomous agent.In a preferred embodiment, agent model data may be obtained fromautonomous agent data and S230 preferably functions to use the agentmodel data to perform a search within the granular mutual observationdata for the subject autonomous agent within a specific period ortemporal window in which it is known or has been determined (as in S220)that the subject autonomous agent may be positioned within afield-of-sensing or field-of-observation of a subject infrastructuredevice.

In this first implementation, S230 may function to perform a featurematching based on the agent model data. In such implementation, S230 mayfunction to use agent model data to compute an alignment betweenobserved features (preferably, within the granular mutual observationdata) and expected features of the subject autonomous agent and thereby,estimate a position of the subject autonomous agent from the perspectiveof the subject infrastructure device. The feature data may includephysical features of the autonomous agent, determined at a time ofmanufacture of the autonomous agent. Additionally, or alternatively, thefeature data may include one or more uniquely identifying physicalfeatures (e.g., a marker, a tag, emblems, etc.) augmented to theautonomous agent data post-manufacture of the autonomous agent. As anexample, a localization circuit or the like may function to perform asearch in a local geographic region from a view point of a subjectautonomous agent (and/or the infrastructure device) for data thatmatches an expected model and/or feature data of the autonomous agent.

2.4 Dissonance Identification/Calibration Error Detection

The method can include S240, which includes identifying dissonance inobservation data from a perspective of a subject infrastructure device,may function to identify instances in which there is a differencebetween observed data (or actual observation data) of the subjectinfrastructure device and expected observation data. Specifically, insome embodiments, S240 may function to identify or detect when theobserved data relating to an observation of a subject autonomous agentdoes not match or may be misaligned with expected observation data ofthe subject autonomous agent.

S240 is preferably performed in response to S230, but can additionallyor alternatively be performed continuously throughout the method 200, inplace of any or all of the above processes, in absence of any or all ofthe above processes, in response to a trigger, once during the method200, and/or at any other suitable times. Further additionally oralternatively, the method 200 can be performed in absence of S240 (e.g.,upon determining that there is no dissonance).

In one embodiment, S240 may function to identify calibration error bycomputing a mismatch value between the observed data and the expectedobservation data of the subject autonomous agent. In one implementation,S240 may function to compute a mismatch value based on a calculated(average) distance between one or more features of the subjectautonomous agent in the observed data and the same one or more featuresof the subject autonomous agent in the expected observation data of theautonomous agent. In a variant of this implementation, S240 may functionto compute a mismatch value based on a computed difference between anactual position of the autonomous agent as observed by a subjectinfrastructure device and an expected position of the subject autonomousagent.

Accordingly, S240 may function to determine that calibration error in asubject infrastructure device exists if a calculated mismatch value forthe subject infrastructure device satisfies and/or exceeds a (first)calibration error threshold. In some embodiments, the calibration errorthreshold may be set based on one or more of a type of infrastructuredevice, a type of calibration error, and/or the like. Additionally, oralternatively, the calibration error threshold may be dynamicallyadjusted based on environmental factors (e.g., expected shifts incalibration due to environment), prior calibration error in a giveninfrastructure device (e.g., previously known or detected calibrationerror), and/or the like. In such embodiments, S240 may function toautomatically and/or dynamically adjust a calibration error thresholdfor a subject infrastructure device based on one or more circumstantialfactors specific to a given infrastructure device.

Additionally, or alternatively, in some embodiments, if a computedmismatch value for the subject infrastructure device exceeds a secondcalibration error threshold that exceeds the first calibration errorthreshold, S240 may function to automatically trigger a requirementand/or notification for manual intervention and/or recalibration. Forinstance, in some embodiments, a computed mismatch value may besufficiently high (i.e., satisfies or exceeds the second calibrationerror threshold) such that the computed error in or uncalibration of asubject infrastructure device cannot be adequately repaired withoutmanual intervention.

In another embodiment, S240 may function to compute calibration errorbased on a computed alignment of observed data and expected observationdata. In such embodiments, S240 may function to compute calibrationcorrection values based on a forced fit and/or best fit alignmentbetween the observed data and the expected observation data. Forinstance, S240 may function to bring into alignment an observed data ofa subject autonomous agent with expected observation data of the subjectautonomous agent. In computing the alignment, S240 may function totranslate, rotation, and/or otherwise move data (image data or the like)from the observed data to match the expected observation data.Accordingly, S240 may function to generate calibration correction valuesbased on the amount and/or value for each type of adjustment made toachieve the force fit or best fit between the observed data and theexpected observation data of the autonomous agent.

S240 can optionally additionally or alternatively include determining asource of the calibration error (e.g., autonomous agent, infrastructuredevice, particular sensor of an autonomous agent and/or infrastructuredevice, etc.) and/or any other features associated with the calibrationerror (e.g., as shown in FIG. 7 ), such as, but not limited to: a timeat which the calibration error occurred; a cause of the calibrationerror (e.g., physical bumping of sensor and/or device, sensor drift,environmental wear, etc.); a previous calibration error; a predictedcalibration error; and/or any other suitable features.

In some variations, for instance, a source of calibration error can bedetermined based on aggregated information. The aggregated informationcan be historical information associated with a particular systemcomponent (e.g., autonomous agent, infrastructure device, etc.) and/orparticular sensor type (e.g., camera, lidar, radar, etc.) or particularsensor (e.g., right side door camera) of the system component;information aggregated over multiple system components (e.g., multipleautonomous agents, multiple infrastructure devices, etc.) and/ormultiple sensors of the same component (e.g., same type of sensor suchas multiple cameras, different types of sensors such camera and radar,etc.) and/or multiple sensors of different components (e.g., same sensortype among different autonomous agents, same sensor type among differentinfrastructure devices, etc.), and/or any other suitable information.

In a first set of specific examples, for instance, S240 includesaggregating information from other infrastructure devices, such that theparticular infrastructure device information can be compared with these.If a detected calibration error persists among different infrastructuredevices and the same autonomous agent, then it can be determined thatthe autonomous agent is at least partially responsible for thecalibration error.

In a second set of specific examples additional or alternative to thosedescribed above, S240 includes aggregating information from otherautonomous agents, such that the particular autonomous agent informationcan be compared with these. If a detected calibration error persistsamong different autonomous agents and the same infrastructure device,then it can be determined that the infrastructure device is at leastpartially responsible for the calibration error.

In a third set of specific examples additional or alternative to thosedescribed above, S240 can include comparing current data with historicaldata associated with that component or a separate component. Forinstance, if a calibration error is detected between a first and secondcomponent, and a calibration error was not detected in a previousinteraction of the first component (e.g., with a third component), butwas detected in a previous interaction of the second component (e.g.,with a fourth component), then it can be determined that the calibrationerror can be at least partially attributed to the second component.

2.5 Recalibration

The method can include S250, which includes recalibrating a subjectinfrastructure device, may function to generate new calibrationparameters (e.g., as shown in FIG. 4 ) based on a mismatch value(s)and/or a correction value(s) for the subject infrastructure device andcommunicate the new calibration parameters to the subject infrastructuredevice. The new calibration parameters preferably comprise a series ofparameters computed based on one or more of the mismatch value and/orthe correction value that, when applied to a subject infrastructuredevice, updates a calibration model and/or existing calibrationparameters of the subject infrastructure device.

S250 is preferably performed in response to S240, but can additionallyor alternatively be performed continuously throughout the method 200, inplace of any or all of the above processes, in absence of any or all ofthe above processes, in response to a trigger, once during the method200, and/or at any other suitable times. Further additionally oralternatively, the method 200 can be performed in absence of S250 (e.g.,upon determining that there is no error in calibration).

In a first implementation, S250 may function to generate new calibrationparameters that include a new calibration model for a subjectinfrastructure device. In this first implementation, S250 may functionto transmit the new calibration model to the subject infrastructuredevice and the subject infrastructure device may function to update itcalibration parameters (recalibrate) by replacing a prior calibrationmodel with the new calibration model. In a variant of this firstimplementation, S250 may function to generate new calibration parametersthat may be used to update a prior calibration model rather thanreplacing the prior calibration model in its entirety.

In a second implementation additional or alternative to the above, S250may function to generate new calibration parameters that include selectcalibration parameter values for one or more specific features of thesubject infrastructure device. In this second implementation, S250 mayfunction to transmit the select calibration parameter values to thesubject infrastructure device and causing the subject infrastructuredevice to update calibration parameters with the select calibrationparameter values. For instance, in some embodiments, a subjectinfrastructure device may include an articulating sensor for whichselect calibration parameter values may be generated that may functionto recalibrate or adjust one or more position data values, such as arotational position or movement, an articulation position or path, anangle position or amount, and/or translation position or movement of thearticulating sensor. In this same instance, S250 may function to updatea physical position of a subject infrastructure device by providing newcalibration parameter values for controlling a servo motor or the likethat can physically reposition one or more features of the subjectinfrastructure device.

It shall be noted that the generation of new calibration parametervalues may additionally, or alternatively, depend on an infrastructuredevice type, model, and/or condition.

In a third implementation additional or alternative to those above, S250can leverage information collected and/or discovered in any or all ofthe above processes to determine (e.g., generate new, update, select,etc.) calibration parameters based on a knowledge of any or all of: thesource of the calibration error (e.g., a particular sensor type, aparticular sensor of the infrastructure device, a particular sensor ofthe autonomous agent, etc.), the cause of the calibration error (e.g.,intrinsic calibration error, extrinsic calibration error, etc.), afeature of the calibration error (e.g., magnitude, time at which thecalibration error occurred, etc.), and/or any other suitableinformation.

3.0 Variations

In a first variation of the system 100 (e.g., as shown in FIG. 1 , asshown in FIG. 2 , etc.), the system includes an autonomous agent 110,wherein the autonomous agent is in the form of an autonomous vehicle; aplurality of infrastructure devices 120, wherein the plurality ofinfrastructure devices are arranged proximal to a route being traversedor to be traversed by the vehicle. In a specific set of examples, eachof the infrastructure devices includes a camera configured to detect theautonomous agent, and optionally additional features, within a region(e.g., wherein the camera lens is chosen based on the size and/or shapeof this region), and wherein the autonomous agent includes at least acombination of cameras, radar, and LiDAR sensors. In another set ofspecific examples, each of the set of infrastructure devices include aset of one or more LiDAR sensors.

In a first variation of the method 200 (e.g., as shown in FIG. 5 , asshown in FIG. 6 , etc.), the method 200 includes: collecting observationinformation from a set of one or more data sources S210, whereinobservation information including position information is collected fromsensors of both the autonomous agent(s) and the infrastructure device(s)along the route of the autonomous agent; identifying and/or selectingmutually optimal observation data S220, wherein selecting the mutuallyoptimal observation data includes pairing data from the autonomous agentand an infrastructure device based on the timestamps associated with theinformation, and wherein the mutually optimal observation data includesinformation from the autonomous agent in which the infrastructure deviceis present (e.g., within a field-of-view of the autonomous agent'ssensors) and information from the infrastructure device in which theautonomous agent is present (e.g., within a field-of-view of theinfrastructure device's sensors); optionally specifically localizing asubject autonomous agent based on granular mutually optimal observationdata S230, wherein the granular mutually optimal observation data isdetermined by removing noise and/or extraneous information (e.g.,information unrelated to the position of the autonomous agent) from themutually optimal observation data; identifying a dissonance inobservation data from a perspective of a subject infrastructure deviceS240, wherein identifying the dissonance can include identifying amismatch value in a location parameter (e.g., location of the autonomousagent) and determining that the mismatch value exceeds a predeterminedthreshold; in the event that the mismatch value exceeds a predeterminedthreshold, determining that a calibration error has occurred; and inresponse to determining that the calibration error has occurred,transmitting a calibration update (e.g., recalibration) to a relevantsource of the calibration error.

Additionally, the method 200 can include receiving health statusinformation from an infrastructure device and identifying a state ofcalibration of the infrastructure device based on the health statusinformation. This can be performed prior to any or all of the remainingprocesses of the method, in place of any or all of the remainingprocesses (e.g., upon determining that the infrastructure device isproperly calibrated, upon determining that the infrastructure device isin need of calibration, etc.), during any or all of the remainingprocesses of the method, after any or all of the remaining processes ofthe method, and/or at any other suitable time(s).

Additionally or alternatively, the method 200 can include aggregatinginformation, which can function to determine a source of the calibrationerror. In specific examples, for instance, any or all of the followinginformation can be compared: current information with historicalinformation (e.g., autonomous agent current information with previousautonomous agent information, infrastructure device current informationwith previous infrastructure device information, etc.); other deviceinformation (e.g., other autonomous agent information, otherinfrastructure device information, etc.); and/or any other suitableinformation.

The systems and methods of the preferred embodiments and variationsthereof can be embodied and/or implemented at least in part as a machineconfigured to receive a computer-readable medium storingcomputer-readable instructions. The instructions are preferably executedby computer-executable components preferably integrated with the systemand one or more portions of the processors and/or the controllers. Thecomputer-readable medium can be stored on any suitable computer-readablemedia such as RAMs, ROMs, flash memory, EEPROMs, optical devices (CD orDVD), hard drives, floppy drives, or any suitable device. Thecomputer-executable component is preferably a general or applicationspecific processor, but any suitable dedicated hardware orhardware/firmware combination device can alternatively or additionallyexecute the instructions.

Although omitted for conciseness, the preferred embodiments includeevery combination and permutation of the implementations of the systemsand methods described herein.

As a person skilled in the art will recognize from the previous detaileddescription and from the figures and claims, modifications and changescan be made to the preferred embodiments of the invention withoutdeparting from the scope of this invention defined in the followingclaims.

1. A method compromising: receiving a first set of observation data,collected with a sensor onboard an autonomous agent, and a second set ofobservation data, collected with a secondary device; determining anassociation between the first and second sets of observation data;localizing the autonomous agent based on the first and second sets ofobservation data; determining a calibration error based on theassociation and the localization of the autonomous agent; and updating astate of calibration based on the calibration error.
 2. The method ofclaim 1, wherein the association comprises a spatiotemporal association.3. The method of claim 1, further comprising: determining a source ofthe calibration error based on a third set of observation data, whereinthe state of calibration is updated based on the source of thecalibration error.
 4. The method of claim 1, further comprising: basedon the association, determining mutually optimal observation data forthe first and second sets of observation data, wherein the calibrationerror is determined with the mutually optimal observation data.
 5. Themethod of claim 4, wherein determining the granular mutually optimalobservation data further comprises removing noise from the mutuallyoptimal observation data.
 6. The method of claim 1, wherein theassociation comprises at least one granular instance of coincidentmutual observability between the first and second sets of observationdata.
 7. The method of claim 6, wherein the autonomous agent islocalized with joint observation data, from both the first and secondsets of observation data, which corresponds to the at least one granularinstance of coincident mutual observability.
 8. The method of claim 1,further comprising: determining a dissonance between the first andsecond data sets based on the association, wherein the calibration erroris determined using the dissonance.
 9. The method of claim 1, whereinupdating the state of calibration comprises updating the state ofcalibration of the sensor.
 10. The method of claim 1, wherein updatingthe state of calibration comprises updating the state of calibration ofthe secondary device.
 11. The method of claim 1, wherein the secondarydevice is remote from the autonomous agent.
 12. The method of claim 11,wherein the secondary device is fixed infrastructure within anenvironment of the autonomous agent.
 13. The method of claim 1, whereinthe secondary device comprises a field of sensing, the second set ofobservation data associated with the field of sensing, wherein theautonomous agent is arranged within the field of sensing.
 14. The methodof claim 1, further comprising: based on the updated state ofcalibration, triggering a recalibration routine to update a set ofcalibration parameters for the sensor or the secondary device.
 15. Asensor device comprising a state of calibration and configured to outputthe state of calibration, the state of calibration determined by:determining a first set of observation data with the sensor device;determining a second set of observation data with a sensor onboard avehicle; determining an association between the first and second sets ofobservation data; localizing the vehicle agent based on the first andsecond sets of observation data; determining a calibration error basedon the association and the localization of the vehicle agent; anddetermining the state of calibration based on the calibration error. 16.The sensor device of claim 15, wherein the association comprises aspatiotemporal association.
 17. The sensor device of claim 15, whereinthe state of calibration is further determined by: determining a sourceof the calibration error based on a third set of observation data,wherein the state of calibration is updated based on the source of thecalibration error.
 18. The sensor device of claim 15, wherein the stateof calibration is further determined by: based on the association,determining mutually optimal observation data for the first and secondsets of observation data, wherein the calibration error is determinedwith the mutually optimal observation data.
 19. A non-transitorycomputer readable medium comprising software instructions that, whenexecuted by a computing system, cause the computing system to perform:determining a first set of observation data, collected with a sensoronboard an autonomous agent, and a second set of observation data,collected with a secondary device; determining an association betweenthe first and second sets of observation data; localizing the autonomousagent based on the first and second sets of observation data;determining a calibration error based on the association and thelocalization of the autonomous agent; and updating a state ofcalibration based on the calibration error.
 20. The non-transitorycomputer readable medium of claim 19, wherein the computing system isonboard the autonomous agent.